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Introduction 


Current  R&D  Requirements:  Debriefing  Distributed  Simulation-Based 
Exercises 

-  Team  Performance  Assessment  &  Diagnosis  Technologies 

-  After  Action  Review  Tools 

Presentation:  Validation  of  a  Measure  for  Assessing  and  Diagnosing 
Team  Tactical  Decision  Making 

Background  &  Approach 

-  Tactical  Decision  Making  Under  Stress  (TADMUS)  6.2 

-  Context:  Combat  Information  Center  Teams  In  Air  Defense  Operations 

-  Objective:  Enhance  The  Quality  Of  Decision-making  In  High  Stress 
Environments  Via: 

•  Phase  1  Products: 

+  Decision  Support  Tools  &  Individual  and  Team  Training  Principles 

•  Phase  2  Products: 

+  Principles  for  Integrating  Training  and  Decision  Support 


Focus:  Air  Defense  Warfare  Team 
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Combat  Display  Overload 


£ 


Displays  &  Training  Don’t  Support 
Managing  the  Tactical  Task 

Limitations  in  Memory 

•  Forgetting  or  mixing  up  numbers  assigned  to  radar  contacts 

•  Forgetting  or  confusing  track  kinematic  data  such  as  bearing, 
altitude,  or  range 

Memory  Support  Tools:  Decision  Biases 

•  Persevering  with  incorrect  threat  assessments  regardless  of  new 
information 

•  Not  considering  all  alternatives  or  hypotheses 

Memory  Demands 

•  Attention  attenuation-competing  cognitive  tasks  &  team 
coordination  requirements 

•  Forgetting  team  communications 

•  Not  attending  to  team  coordination  requirements 

Team  Coordination  Requirements 


Decision  Making  Dimensions 
(Marshall  et  al.,  1995) 


IDENTIFICATION 

•  Recognizing  the  Problem 

•  Effective  use  of  Pattern 
Recognition  Strategies  using 
Track  Profile  Information 


r 


EXECUTION 

•  Effective  and  Timely 
Implementation  of  Plans 

•  Knowing  Who  Should  Perform 
the  Required  Actions 


ELABORATION 

Interpreting  the  Situation 
Recognizing  Constraints 

CODA  *  Threats 

Loop 


PLANNING 

•  Knowing  What  to  Do  and  When 
To  Do  It 
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DECISION-MAKING  DIMENSIONS  & 
TADMUS  Decision  Support  System 


Pianning 

Identification 

Knowing  What  to  Do 

Recognizing  the  Problem 

and  When  To  Do  It 

Effective  use  of  Pattern 
Recognition  Strategies 
using  Track  Profile 

Pianning  & 
Execution 

Elaboration 

Interpreting  the 
Situation 

Information 

Effective  and  Timely 

Recognizing 

Implementation  of 

Constraints 

Plans 

Prioritizing  Threats 

Identification 


identification 
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TADMUS  DECISION  SUPPORT  SYSTEM  (DSS) 

Morrison  et  al.  1998 
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TADMUS  Training  Strategies 

(Cannon-Bowers  &  Salas,  1998) 


Scenario-Based  Training/Event  Based  Approach  (Johnston  et  al.,  1998) 
Identification/Elaboration  Cognitive  Processes 

-  Critical  Thinking  Training-Know  Thyself...  (Cohen  et  al.,  1998) 

Planning  &  Execution  Team  Tasks:  Knowing  Who  Should  Perform  The  Required 
Actions 

-  Team  Dimensional  Training  (Smith-Jentsch  et  al.,  1998) 

•  Team  Leader  Training-The  Blind  Pass...  (Tannenbaum  etal.,  1998) 

•  Team  Coordination  Training-He  Ain't  Heavy... (Serfaty  et  al.,  1998) 

•  Team  Self-Correction-  Replay  in  the  Bar... (Smith-Jentsch  et  al.,  1998) 

-  Cross  Training-Walking  in  Each  Other's  Shoes... (Blickensderfer  et  al.,  1998) 


PERFORMANCE  MEASUREMENT 

SCHEME 


INDIVIDUAL 

P 

-  Decision  Making  Processes 
^  -  Task  Strategy 

^  -  Information  Seeking 

^  Behavior  Observation  Booklet 

5 

Critical  Thinkina  Strateoies 

TEAM 

-  Coordination  Behaviors 

-  Communication  Flow 

-  Team  Strategies 

Air  Warfare  Team  Observation  Measure 

Anticipation  Ratio 

0 

y  -  Accuracy 

-  Mission  Effectiveness 

j  -  Timeliness 

-  Team  Level  Timeliness  &  Accuracy 

C  -  Decision  Biases 

-  Error  Propagation 

0 

^  Seauenced  Actions  &  Latencies  Index 

Air  Warfare  Team  Performance  Index 

^  Vocalized  Priorities  (SA) 

DESCRIBE,  EVALUATE,  DIAGNOSE  BEHAVIOR 
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TADMUS  Final  Demonstration 


•  Intervention:  Combined  Impact  of  Training  and  DSS  on  Team 
Performance 

•  Hypotheses: 

-  Less  Perceived  Stress 

-  Better  Teamwork  Performance 

-  Better  Critical  Thinking  Skills 

-  Better  Tactical  Performance 

-  Teamwork  is  Related  to  Tactical  Performance 

•  Integrated  Training  Approach  Via  Scenario-Based  Training 

-  Individual  Skills  Training 

•  Critical  Thinking  Skills  Training  (PC-Based) 

•  Decision  Support  System  (DSS)  Tutorial 

-  Team  Skills  Training 

•  Team  Dimensional  Training  (TDT): 

+  Facilitated  Team  Self-Correction  with  Event-Based  Scenarios 

•  Cross  Training: 

Utilized  DSS  for  AAR  to  Facilitate  Discussion  of  Decision  Making  Processes 


+ 


TADMUS  Final  Demonstration 

Approach 

Participants:  Total  of  16  Six-Person  Teams  (Surface  Warfare 
Officer’s  School  Department  Head  Classes)  in  control  (8)  and  in 
experimental  (8)  conditions 

Design:  Multiple  Post-Test  (Arabian  Gulf  Event-Based 
Scenarios)  Counterbalanced 

Task:  Air  Defense  Warfare  (Decision  Making  Evaluation  Facility 
for  Tactical  Teams) 

Individual  and  Team  Measures 

-  Stress:  NASA  TLX 

-  Teamwork:  ATOM 

-  Team  Tactical  Decision  Making  (ATPI) 

-  TAO  Decision  Making  Priorities 

-  Critical  Thinking  Indicators  (under  development) 

Recorded  Team  Voice  Comms,  Created  Transcripts 


DEFTT  Lab  at  SWOS 


Debrief/AAR  Using  the  DSS 
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Research  Problem: 

Need  Diagnostic  Team  Performance  Assessment 
Tools  to  Identify  Critical  Team  Knowledge  and  Skill  Deficits 

Hypotheses: 

Measure  of  Team  Taskwork,  based  on  Marshall  et  al.  model,  will  increase  diagnosticity  of 
team  tactical  decision  making  performance  (Paris  et  al.,  2001) 


Using  Taskwork  &  Teamwork  measures  will  increase  the  diagnosticity  of  team  performance 
for  AAR  (Johnston,  Freeman,  &  Serfaty,  2003) 


ID  &  Prioritization 

Actions 

Planning  &  Execution 

Actions 

Plan  to  Engage/Engage 
(What  is  typically  discussed) 

•  Recognizing  the  Problem 
(ID) 

•  Effective  use  of  Pattern 
Recognition  Strategies  using 
Track  Profile  Information  (ID) 

•  Interpreting  the  Situation 
(ELAB) 

•  Recognizing  Constraints 
(ELAB) 

•  Prioritizing  Threats  (ELAB) 

•  Knowing  What  to  Do  and 
When  To  Do  It  (Planning) 

•  Effective  and  Timely 
Implementation  of  Plans 
(Execution) 

•  Knowinp  Who  Should 

Perform  the  Reauired 

Actions  (Execution) 

•  NMETLs 

•  e.g..  Proper  Weapons 
Employment 

•  Using  proper  weapons 
engagement  procedures 
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Team  Taskwork:  Detect-to-Engage  Sequence 

(Paris,  Johnston,  &  Reeves,  1998) 


IDENTIFICATION 


Detect 

Entity  Type  ID 


t 


EXECUTION 

Query 

Final  Warning 
iiiuminate 

Cover  With  Weapons 
Engage 


ELABORATiON 


•  Threat  iD 
Threat  Priority 


eoDA 

Loop 


i 


PLANNiNG 

Query 

Final  Warning 
iiiuminate 

Cover  With  Weapons 
Engage 
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Air  Warfare  Team  Performance  Index 
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ATPI  Analysis 


•  Created  6  ATPIs 

-Alpha  &Tango  training  scenarios 

-Bravo,  Charlie,  Delta,  India  Experiment  Scenarios 

•  Two  SMEs  used  transcripts  of  team  communications  (obtained  very  close 
agreement) 


Summed 
Instances  of 
Actions  Across 

4  Test 

Scenarios 

Summed  ID  & 
Elaboration 

Actions 

Summed  Planning 
&  Execution 

Actions 

Plan  to  Engage 
and 

Engage  Actions 

Tactical  A/C 

•  Correct 

•  Correct,  But  Late 

•  Incorrect 
(Wrong/Missing) 

•  Correct 

•  Correct,  But  Late 

•  Incorrect 
(Wrong/Missing) 

•  Correct 

•  Correct,  But  Late 

•  Incorrect 
(Wrong/Missing) 

Commercial 

A/C 

•  Correct 

•  Correct,  But  Late 

•  Incorrect 
(Wrong/Missing) 

•  Correct 

•  Correct,  But  Late 

•  Incorrect 
(Wrong/Missing) 

Incorrect 

TEAMWORK  DIMENSIONS:  PROCESSES 
(Smith-Jentsch  et  al.,  1998) 


INFO  EXCHANGE 

•  Communication  that  promotes  a  team 
awareness  of  the  surrounding 
environment,  both  internal  and  external 
to  the  team. 

•  Timely  and  accurate  reporting  of 
deviations  and/or  potential  problems 


SUPPORTING  BEHAVIOR 


•  Monitoring  the  activities  of  other  team 
members,  taking  action  to  correct  errors, 
giving  and  receiving  feedback  in  a 
nondefensive  manner 

•  Providing  and  seeking  assistance  or 
backup  when  needed. 


COMMUNICATION 

•  Clear  and  efficient  exchange  of  relevant 
information. 

•  Using  proper  terminology,  standard 
procedures  for  external  communications, 
and  an  appropriate  tone  of  voice. 


INITIATIVE/LEADERSHIP 

•  Providing  needed  guidance  to  other 
team  members;  helping  team  members 
focus  their  activities  appropriately  and 
anticipate  tasks  that  should  be 
performed; 

•  Providing  instruction  to  other  team 
members  to  enable  team  to  perform  or 
complete  their  tasks.  Any  team  member 
can  perform  initiative  /  leadership 
functions. 
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Air  Warfare  Team  Observation  Measure  (ATOM) 
Example  of  Information  Exchange  for  Evaluation 


Frequency  of  Seeking  sources  -  How  many  times  did 
the  team  members  proactively  ask  for  information  from 
multiple  sources  in  order  to  establish  an  accurate 
assessment  of  the  situation.  These  sources  may  be 
internal  or  external  to  the  team  and  may  include  written 
documentation. 

None  =  Seeking  information  is  a  reai  weakness  for  this 
team 

1-2  times=  Seeking  info  is  adequate  for  this  team 

3  or  more  times=  Seeking  information  is  a  Strength  for  this 
team 


Air  Warfare  Team  Observation  Measure  (ATOM) 
Example  of  Information  Exchange  for  Training 


Anchored  Scale  for  Seeking  sources  -  Proactively 
asking  for  information  from  multiple  sources  in  order  to 
establish  an  accurate  assessment  of  the  situation. 
These  sources  may  be  internal  or  external  to  the  team 
and  may  include  written  documentation. 


1  2  3  4  5 


Seeking  information 
a  real  weakness 
for  this  team. 


Seeking  information  is 
is  a  real  strength  for 
this  team. 
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ATPI  Outcomes 


Number  of 

Tactical 

Engagements 

Correct 

Plan  to 
Engage 
lAW  ROE 

Correct 

Engage 

lAW 

ROE 

Plan  to 

Engage  not 
lAW  ROE 

Engage  not 
lAW  ROE 

Control 

Tactical 

Commercial 

•  7 

•  N/A 

•  6 

•  N/A 

•10  (9  teams) 

•  1 

•  7  (6  teams) 

•  0 

Expt’l 

Tactical 

Commercial 

•  7 

•  N/A 

•  6 

•  N/A 

•  8  (7  teams) 

•  0 

•  6  (5  teams) 

•  0 
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ATPI:  Team  Task  Processes 


ID  &  Elaboration 

Tactical  Actions  lAW 
ROE 

Percent  of  Possible 
Correct  +  Correct,  But 
Late 

Planning  & 

Execution  Actions 
lAW  ROE 

Percent  of  Possible 
Correct  +  Correct, 

But  Late 

Number  of 
Instances  of 
Incorrect 
Planning  & 
Execution 
Actions 

Control 

Tactical 

•  83  (range  78-89) 

•  30  (range  25-41)* 

•  0 

Commercial 

•  68  (range  48-83) 

•  19  (range  0-45) 

•16  (6  teams) 

Expt’l 

Tactical 

•  85  (range  80-88) 

•  36  (range  22-47)* 

•  9  (6  teams) 

Commercial 

•  54  (range  29-71) 

•  22  (range  5-45) 

•  11  (4  teams) 

*  Problem  appears  to  be  NOT  taking  action 
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Initial  Results: 

Relationship  of  Teamwork  and  Taskwork 


1^^ 


•  DSS/Trained  Teams  Achieved  Significantly  Better  (p<.001)  Teamwork 
Performance  (about  1 1  %  better  after  each  scenario),  than  teams  in  the  control 
condition 

•  In  Preparation  For  Potentially  Hostile  Tactical  Aircraft,  DSS/Trained  Teams 

-  Showed  a  trend  in  performing  more  correct  actions  across  ID/ELAB/Planning/Exec 
(p=.07),  but  no  correlation  with  teamwork  (r=.260,  n.s.)  + 

-  Showed  a  trend  in  waiting  longer  (a  few  seconds)  across  ID/ELAB/Planning/Exec, 
but  performed  more  correct  actions  (much  greater  variability  in  performance  than 
control  condition)  (p<.1).  Correlation  with  teamwork  was  .418,  but  n.s.+ 

-  Waited  longer  (a  few  seconds),  but  performed  significantly  more  correct 
planning/execution  actions  (p<.005).  Significant  correlation  with  teamwork 
performance  (r=.532*)+ 


+  Pooled  Within  Groups  Correlation  (Across  Both  Conditions)  b/w  Teamwork  and  ATPI 
metrics  (.426*,  p<.05,  df=14) 


27 


After  Action  Review 


ID  &  Prioritization 

Actions 

Planning  &  Execution 

Actions 

Plan  to  Engage/Engage 
(Currently  what  gets 
discussed) 

•  Recognizing  the  Problem 

•  Effective  use  of  Pattern 
Recognition  Strategies  using 
Track  Profile  Information 

•  Interpreting  the  Situation 

•  Recognizing  Constraints 

•  Prioritizing  Threats 

•  Knowing  What  to  Do  and 
When  To  Do  It 

•  Effective  and  Timely 
Implementation  of  Plans 

•  Knowing  Who  Should 
Perform  the  Required 
Actions — T  eamwork 
Implicated  Here 

•  NMETLs 

•  e.g.,  Proper  Weapons 
Employment 

•  e.g.,  Using  proper 
weapons  engagement 
procedures 

•  Proper  Operation  of  Radar 
System 

Team  Self  Correction  in  AAR  Can  Focus  on  Planning  &  Execution 
Actions,  to  Rapidly  Narrowing  the  Focus  on  Specific  Knowledge  and  Skill 
Deficiencies 
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•  GUI  Concepts  for  AAR  on  Team  Decision  Making 

-  Replay  of  Training  Ground  Truth 

-  Expected  Team  Performance  Compared  with  Actual  Team  Performance 

-  Voice  Reports  and  Watchstander  Actions  Incorporated  Into  Replay 

-  Review  of  Events  Walk  Through  to  Address: 

•  Team  ID/ELAB  Processes 

+  Basis  for  Assessment 
+  Track  Priorities 

•  Team  Planning/Execution  Processes  Based  on  Task  Manager  Results 
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